{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 推荐系统组队学习  协同过滤算法\n",
    "\n",
    "根据用户之前的喜好以及其他兴趣相近的用户的选择来给用户推荐物品(基于对用户历史行为数据的挖掘发现用户的喜好偏向， 并预测用户可能喜好的产品进行推荐)，一般是仅仅基于用户的行为数据（评价、购买、下载等）, 而不依赖于项的任何附加信息（物品自身特征）或者用户的任何附加信息（年龄， 性别等）。\n",
    "\n",
    "分类：\n",
    "基于用户的协同过滤算法(UserCF): 给用户推荐和他兴趣相似的其他用户喜欢的产品  \n",
    "基于物品的协同过滤算法(ItemCF): 给用户推荐和他之前喜欢的物品相似的物品\n",
    "\n",
    "**重要的步骤之一就是计算用户和用户或者物品和物品之间的相似度**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 相似性度量方法\n",
    "\n",
    "### 1.杰卡德(Jaccard)相似系数\n",
    "\n",
    "两个用户u和v交互商品交集的数量占这两个用户交互商品并集的数量的比例，称为两个集合的杰卡德相似系数，用符号 \n",
    "$sim_{uv}$表示，其中N(u),N(v)分别表示用户u和用户v交互商品的集合。\n",
    "\n",
    "$$sim_{uv}=\\frac{|N(u) \\cap N(v)|}{\\sqrt{|N(u)| \\cup|N(v)|}}$$\n",
    "\n",
    "也就是交集与并集数量之比\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.余弦相似度\n",
    "\n",
    "余弦相似度衡量了两个向量的夹角，夹角越小越相似。首先从集合的角度描述余弦相似度，相比于Jaccard公式来说就是分母有差异，不是两个用户交互商品的并集的数量，而是两个用户分别交互的商品数量的乘积，公式如下：\n",
    "\n",
    "$$sim_{uv}=\\frac{|N(u)| \\cap |N(v)|}{\\sqrt{|N(u)|\\cdot|N(v)|}}$$\n",
    "\n",
    "从向量的角度进行描述，令矩阵A为用户-商品交互矩阵(因为是TopN推荐并不需要用户对物品的评分，只需要知道用户对商品是否有交互就行)，即矩阵的每一行表示一个用户对所有商品的交互情况，有交互的商品值为1没有交互的商品值为0，矩阵的列表示所有商品。若用户和商品数量分别为m,n的话，交互矩阵A就是一个m行n列的矩阵。此时用户的相似度可以表示为:\n",
    "\n",
    "$$ sim_{uv} = cos(u,v) =\\frac{u\\cdot v}{|u|\\cdot |v|}$$\n",
    "\n",
    "避免存储很大的稀疏矩阵，在计算用户相似度的时候一般会采用集合的方式进行计算。理论上向量之间的相似度计算公式都可以用来计算用户之间的相似度，但是会根据实际的情况选择不同的用户相似度度量方法。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 0.81649658],\n",
       "       [0.81649658, 1.        ]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "i = [1, 0, 0, 0]\n",
    "j = [1, 0.5, 0.5, 0]\n",
    "cosine_similarity([i, j])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.皮尔逊相关系数\n",
    "\n",
    "$r_{ui}$,$r_{vi}$分别表示用户u和用户v对商品i是否有交互,首先对于上述的余弦相似度的计算公式写成求和的形式:\n",
    "\n",
    "$$ sim_{uv} = \\frac{\\sum_i r_{ui}*r_{vi}}{\\sqrt{\\sum_i r_{ui}^2}\\sqrt{\\sum_i r_{vi}^2}}$$\n",
    "\n",
    "其中$r_{ui},r_{vi}$分别表示用户u和用户v对商品i是否有交互(或者具体的评分值)，$\\bar r_u, \\bar r_v$分别表示用户u和用户v交互的所有商品交互数量或者具体评分的平均值。\n",
    "\n",
    "$$sim(u,v)=\\frac{\\sum_{i\\in I}(r_{ui}-\\bar r_u)(r_{vi}-\\bar r_v)}{\\sqrt{\\sum_{i\\in I }(r_{ui}-\\bar r_u)^2}\\sqrt{\\sum_{i\\in I }(r_{vi}-\\bar r_v)^2}}$$\n",
    "\n",
    "相比余弦相似度，皮尔逊相关系数通过使用用户的平均分对各独立评分进行修正，减小了用户评分偏置的影响。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.816496580927726, 0.18350341907227397)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import pearsonr\n",
    "\n",
    "i = [1, 0, 0, 0]\n",
    "j = [1, 0.5, 0.5, 0]\n",
    "pearsonr(i, j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.基于用户的协同过滤\n",
    "\n",
    "思想：当一个用户A需要个性化推荐的时候， 我们可以先找到和他有相似兴趣的其他用户， 然后把那些用户喜欢的， 而用户A没有听说过的物品推荐给A。\n",
    "\n",
    "步骤：   \n",
    "1.找到和目标用户兴趣相似的集合：基于前面给出的相似性度量的方法找出与目标用户兴趣相似的用户    \n",
    "2.找到这个集合中的用户喜欢的， 且目标用户没有听说过的物品推荐给目标用户：依赖于目标用户对相似用户喜欢的物品的一个喜好程度  \n",
    "\n",
    "|  |物品1|物品2|物品3|物品4|物品5|\n",
    "|--|--|--|--|--|--|\n",
    "Alice|5|3|4|4|?\n",
    "用户1|3|1|2|3|3\n",
    "用户2|4|3|4|3|5\n",
    "用户3|3|3|1|5|4\n",
    "用户4|1|5|5|2|1\n",
    "\n",
    "给用户推荐物品的过程可以形象化为一个猜测用户对商品进行打分的任务，上面表格里面是5个用户对于5件物品的一个打分情况，就可以理解为用户对物品的喜欢程度\n",
    "\n",
    "计算步骤：\n",
    "\n",
    "1.首先根据前面的这些打分情况(或者说已有的用户向量）计算一下Alice和用户1， 2， 3， 4的相似程度， 找出与Alice最相似的n个用户\n",
    "\n",
    "2.据这n个用户对物品5的评分情况和与Alice的相似程度会猜测出Alice对物品5的评分，如果评分比较高的话，就把物品5推荐给用户Alice，否则不推荐。\n",
    "\n",
    "最终结果的预测:  \n",
    "利用用户相似度和相似用户的评价加权平均获得用户的评价预测\n",
    "\n",
    "权重$w_{u,s}$是用户u和用户s的相似度，$R_{s,p}$是用户s对物品p的评分\n",
    "\n",
    "$$R_{\\mathrm{u}, \\mathrm{p}}=\\frac{\\sum_{\\mathrm{s} \\in S}\\left(w_{\\mathrm{u}, \\mathrm{s}} \\cdot R_{\\mathrm{s}, \\mathrm{p}}\\right)}{\\sum_{\\mathrm{s} \\in S} w_{\\mathrm{u}, \\mathrm{s}}}$$\n",
    "\n",
    "还有一种方式如下， 这种方式考虑的更加前面， 依然是用户相似度作为权值， 但后面不单纯的是其他用户对物品的评分， 而是该物品的评分与此用户的所有评分的差值进行加权平均， 这时候考虑到了有的用户内心的评分标准不一的情况， 即有的用户喜欢打高分， 有的用户喜欢打低分的情况。\n",
    "\n",
    "$$P_{i, j}=\\bar{R}_{i}+\\frac{\\sum_{k=1}^{n}\\left(S_{i, k}\\left(R_{k, j}-\\bar{R}_{k}\\right)\\right)}{\\sum_{k=1}^{n} S_{j, k}}$$\n",
    "\n",
    "在获得用户 \n",
    "u\n",
    " 对不同物品的评价预测后， 最终的推荐列表根据预测评分进行排序得到。 至此，基于用户的协同过滤算法的推荐过程完成。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 计算Alice与其他用户的相似度\n",
    "\n",
    "Alice(5,3,4,4)  user1(3,1,2,3) user2(4,3,4,3) user3(3,3,1,5) user4(1,5,,5,2)\n",
    "\n",
    "$$ sim(Alice,user1) = \\frac{5*3 + 3*1 + 4*2 + 4*3 }{sqrt(5*5 + 3*3 + 4*4 + 4*4) + sqrt(3*3 + 1*1 + 2*2 + 3*3)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 0.9753213 , 0.99224264, 0.89072354, 0.79668736],\n",
       "       [0.9753213 , 1.        , 0.94362852, 0.91160719, 0.67478587],\n",
       "       [0.99224264, 0.94362852, 1.        , 0.85280287, 0.85811633],\n",
       "       [0.89072354, 0.91160719, 0.85280287, 1.        , 0.67082039],\n",
       "       [0.79668736, 0.67478587, 0.85811633, 0.67082039, 1.        ]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "users = np.array([[5,3,4,4],[3,1,2,3],[4,3,4,3],[3,3,1,5],[1,5,5,2]])\n",
    "cosine_similarity(users)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.        ,  0.85280287,  0.70710678,  0.        , -0.79211803],\n",
       "       [ 0.85280287,  1.        ,  0.30151134,  0.42640143, -0.88662069],\n",
       "       [ 0.70710678,  0.30151134,  1.        , -0.70710678, -0.14002801],\n",
       "       [ 0.        ,  0.42640143, -0.70710678,  1.        , -0.59408853],\n",
       "       [-0.79211803, -0.88662069, -0.14002801, -0.59408853,  1.        ]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.corrcoef(users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 根据用户对物品5的评分\n",
    "\n",
    "$$P_{Alice, 物品5}=\\bar{R}_{Alice}+\\frac{\\sum_{k=1}^{2}\\left(S_{Alice,user k}\\left(R_{userk, 物品5}-\\bar{R}_{userk}\\right)\\right)}{\\sum_{k=1}^{2} S_{Alice, userk}}=4+\\frac{0.85*(3-2.4)+0.7*(5-3.8)}{0.85+0.7}=4.87$$\n",
    "\n",
    "得到了Alice对物品5的得分是4.87， 根据Alice的打分对物品排个序从大到小： \n",
    "物品1>物品5>物品3=物品4>物品2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算步骤\n",
    "\n",
    "计算用户相似性矩阵、得到前n个相似用户、计算最终得分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#首先定义数据集\n",
    "import pandas as pd\n",
    "def loadData():\n",
    "    items={'A': {1: 5, 2: 3, 3: 4, 4: 3, 5: 1},\n",
    "           'B': {1: 3, 2: 1, 3: 3, 4: 3, 5: 5},\n",
    "           'C': {1: 4, 2: 2, 3: 4, 4: 1, 5: 5},\n",
    "           'D': {1: 4, 2: 3, 3: 3, 4: 5, 5: 2},\n",
    "           'E': {2: 3, 3: 5, 4: 4, 5: 1}\n",
    "          }\n",
    "    users={1: {'A': 5, 'B': 3, 'C': 4, 'D': 4},\n",
    "           2: {'A': 3, 'B': 1, 'C': 2, 'D': 3, 'E': 3},\n",
    "           3: {'A': 4, 'B': 3, 'C': 4, 'D': 3, 'E': 5},\n",
    "           4: {'A': 3, 'B': 3, 'C': 1, 'D': 5, 'E': 4},\n",
    "           5: {'A': 1, 'B': 5, 'C': 5, 'D': 2, 'E': 1}\n",
    "          }\n",
    "    return items,users\n",
    "\n",
    "items, users = loadData()\n",
    "item_df = pd.DataFrame(items).T\n",
    "user_df = pd.DataFrame(users).T \n",
    "\n",
    "#矩阵中，列代表每个用户，行代表每个商品， 值代表用户和用户的相关性\n",
    "#当不是同一个用户的时候， 我们要去遍历物品-用户评分数据， 在里面去找这两个用户同时对该物品评过分的数据放入到这两个用户向量中\n",
    "\n",
    "\"\"\"计算用户相似性矩阵\"\"\"\n",
    "similarity_matrix = pd.DataFrame(np.zeros((len(users), len(users))), index=[1, 2, 3, 4, 5], columns=[1, 2, 3, 4, 5])\n",
    "\n",
    "# 遍历每条用户-物品评分数据\n",
    "for userID in users:\n",
    "    for otheruserId in users:\n",
    "        vec_user = []\n",
    "        vec_otheruser = []\n",
    "        if userID != otheruserId:\n",
    "            for itemId in items:   # 遍历物品-用户评分数据\n",
    "                itemRatings = items[itemId]        # 这也是个字典  每条数据为所有用户对当前物品的评分\n",
    "                if userID in itemRatings and otheruserId in itemRatings:  # 说明两个用户都对该物品评过分\n",
    "                    vec_user.append(itemRatings[userID])\n",
    "                    vec_otheruser.append(itemRatings[otheruserId])\n",
    "            # 这里可以获得相似性矩阵(共现矩阵)\n",
    "            similarity_matrix[userID][otheruserId] = np.corrcoef(np.array(vec_user), np.array(vec_otheruser))[0][1]\n",
    "            #similarity_matrix[userID][otheruserId] = cosine_similarity(np.array(vec_user), np.array(vec_otheruser))[0][1]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.852803</td>\n",
       "      <td>0.707107</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.792118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.852803</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.467707</td>\n",
       "      <td>0.489956</td>\n",
       "      <td>-0.900149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.707107</td>\n",
       "      <td>0.467707</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.161165</td>\n",
       "      <td>-0.466569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.489956</td>\n",
       "      <td>-0.161165</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.641503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-0.792118</td>\n",
       "      <td>-0.900149</td>\n",
       "      <td>-0.466569</td>\n",
       "      <td>-0.641503</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          1         2         3         4         5\n",
       "1  0.000000  0.852803  0.707107  0.000000 -0.792118\n",
       "2  0.852803  0.000000  0.467707  0.489956 -0.900149\n",
       "3  0.707107  0.467707  0.000000 -0.161165 -0.466569\n",
       "4  0.000000  0.489956 -0.161165  0.000000 -0.641503\n",
       "5 -0.792118 -0.900149 -0.466569 -0.641503  0.000000"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "用户Alice对物品5的打分:  4.871979899370592\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.87198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B    C    D        E\n",
       "1  5.0  3.0  4.0  4.0  4.87198\n",
       "2  3.0  1.0  2.0  3.0  3.00000\n",
       "3  4.0  3.0  4.0  3.0  5.00000\n",
       "4  3.0  3.0  1.0  5.0  4.00000\n",
       "5  1.0  5.0  5.0  2.0  1.00000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"计算前n个相似的用户\"\"\"\n",
    "n = 2\n",
    "similarity_users = similarity_matrix[1].sort_values(ascending=False)[:n].index.tolist()    # [2, 3]   也就是用户1和用户2\n",
    "\n",
    "\"\"\"计算最终得分\"\"\"\n",
    "base_score = np.mean(np.array([value for value in users[1].values()]))\n",
    "weighted_scores = 0.\n",
    "corr_values_sum = 0.\n",
    "for user in similarity_users:  # [2, 3]\n",
    "    corr_value = similarity_matrix[1][user]            # 两个用户之间的相似性\n",
    "    mean_user_score = np.mean(np.array([value for value in users[user].values()]))    # 每个用户的打分平均值\n",
    "    weighted_scores += corr_value * (users[user]['E']-mean_user_score)      # 加权分数\n",
    "    corr_values_sum += corr_value\n",
    "final_scores = base_score + weighted_scores / corr_values_sum\n",
    "print('用户Alice对物品5的打分: ', final_scores)\n",
    "user_df.loc[1]['E'] = final_scores\n",
    "user_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UserCF的缺点\n",
    "\n",
    "#### 数据稀疏性：\n",
    "\n",
    "一个大型的电子商务推荐系统一般有非常多的物品，用户可能买的其中不到1%的物品，不同用户之间买的物品重叠性较低，导致算法无法找到一个用户的邻居，即偏好相似的用户。这导致UserCF不适用于那些正反馈获取较困难的应用场景(如酒店预订， 大件商品购买等低频应用)\n",
    "\n",
    "#### 算法扩展性\n",
    "\n",
    "基于用户的协同过滤需要维护用户相似度矩阵以便快速的找出Topn相似用户， 该矩阵的存储开销非常大，存储空间随着用户数量的增加而增加，不适合用户数据量大的情况使用。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基于物品的协同过滤\n",
    "\n",
    "预先根据所有用户的历史偏好数据计算物品之间的相似性，然后把与用户喜欢的物品相类似的物品推荐给用户。比如物品a和c非常相似，因为喜欢a的用户同时也喜欢c，而用户A喜欢a，所以把c推荐给用户A。\n",
    "\n",
    "ItemCF算法并不利用物品的内容属性计算物品之间的相似度， 主要通过分析用户的行为记录计算物品之间的相似度， 该算法认为， 物品a和物品c具有很大的相似度是因为喜欢物品a的用户大都喜欢物品c。\n",
    "\n",
    "步骤：\n",
    "\n",
    "1.计算物品之间的相似度\n",
    "2.根据物品的相似度和用户的历史行为给用户生成推荐列表（购买了该商品的用户也经常购买的其他商品）\n",
    "\n",
    "以Alice为例：\n",
    "\n",
    "|  |物品1|物品2|物品3|物品4|物品5|\n",
    "|--|--|--|--|--|--|\n",
    "Alice|5|3|4|4|?\n",
    "用户1|3|1|2|3|3\n",
    "用户2|4|3|4|3|5\n",
    "用户3|3|3|1|5|4\n",
    "用户4|1|5|5|2|1\n",
    "\n",
    "计算步骤：\n",
    "\n",
    "1.首先计算一下物品5和物品1， 2， 3， 4之间的相似性(它们也是向量的形式， 每一列的值就是它们的向量表示， 因为ItemCF认为物品a和物品c具有很大的相似度是因为喜欢物品a的用户大都喜欢物品c， 所以就可以基于每个用户对该物品的打分或者说喜欢程度来向量化物品)\n",
    "\n",
    "2.找出与物品5最相近的n个物体\n",
    "\n",
    "3.根据Alice对最相近的n个物体的打分去计算对物品5的打分情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>items1</th>\n",
       "      <th>items2</th>\n",
       "      <th>items3</th>\n",
       "      <th>items4</th>\n",
       "      <th>items5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>items1</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.648886</td>\n",
       "      <td>-0.435286</td>\n",
       "      <td>0.473684</td>\n",
       "      <td>0.969458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>items2</th>\n",
       "      <td>-0.648886</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.670820</td>\n",
       "      <td>-0.324443</td>\n",
       "      <td>-0.478091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>items3</th>\n",
       "      <td>-0.435286</td>\n",
       "      <td>0.670820</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.870572</td>\n",
       "      <td>-0.427618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>items4</th>\n",
       "      <td>0.473684</td>\n",
       "      <td>-0.324443</td>\n",
       "      <td>-0.870572</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.581675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>items5</th>\n",
       "      <td>0.969458</td>\n",
       "      <td>-0.478091</td>\n",
       "      <td>-0.427618</td>\n",
       "      <td>0.581675</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          items1    items2    items3    items4    items5\n",
       "items1  1.000000 -0.648886 -0.435286  0.473684  0.969458\n",
       "items2 -0.648886  1.000000  0.670820 -0.324443 -0.478091\n",
       "items3 -0.435286  0.670820  1.000000 -0.870572 -0.427618\n",
       "items4  0.473684 -0.324443 -0.870572  1.000000  0.581675\n",
       "items5  0.969458 -0.478091 -0.427618  0.581675  1.000000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "items = np.array([[3,4,3,1],[1,3,3,5],[2,4,1,5],[3,3,5,2],[3,5,4,1]])\n",
    "cols = ['items'+str(i) for i in range(1,6)]\n",
    "pd.DataFrame(np.corrcoef(items),columns=cols,index=cols)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>items1</th>\n",
       "      <th>items2</th>\n",
       "      <th>items3</th>\n",
       "      <th>items4</th>\n",
       "      <th>items5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>items1</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.738988</td>\n",
       "      <td>0.747667</td>\n",
       "      <td>0.936916</td>\n",
       "      <td>0.994100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>items2</th>\n",
       "      <td>0.738988</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.933564</td>\n",
       "      <td>0.813629</td>\n",
       "      <td>0.738851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>items3</th>\n",
       "      <td>0.747667</td>\n",
       "      <td>0.933564</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.709718</td>\n",
       "      <td>0.722610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>items4</th>\n",
       "      <td>0.936916</td>\n",
       "      <td>0.813629</td>\n",
       "      <td>0.709718</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.939558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>items5</th>\n",
       "      <td>0.994100</td>\n",
       "      <td>0.738851</td>\n",
       "      <td>0.722610</td>\n",
       "      <td>0.939558</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          items1    items2    items3    items4    items5\n",
       "items1  1.000000  0.738988  0.747667  0.936916  0.994100\n",
       "items2  0.738988  1.000000  0.933564  0.813629  0.738851\n",
       "items3  0.747667  0.933564  1.000000  0.709718  0.722610\n",
       "items4  0.936916  0.813629  0.709718  1.000000  0.939558\n",
       "items5  0.994100  0.738851  0.722610  0.939558  1.000000"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cosine_similarity(items),columns=cols,index=cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "找到与物品5最相似的2个物品是item1和item4(n=2)， 下面基于上面的公式计算最终得分：\n",
    "\n",
    "$$ P_{Alice, 物品5}=\\bar{R}_{物品5}+\\frac{\\sum_{k=1}^{2}\\left(S_{物品5,物品 k}\\left(R_{Alice, 物品k}-\\bar{R}_{物品k}\\right)\\right)}{\\sum_{k=1}^{2} S_{物品k, 物品5}}=\\frac{13}{4}+\\frac{0.97*(5-3.2)+0.58*(4-3.4)}{0.97+0.58}=4.6$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: {'A': 5, 'B': 3, 'C': 4, 'D': 4},\n",
       " 2: {'A': 3, 'B': 1, 'C': 2, 'D': 3, 'E': 3},\n",
       " 3: {'A': 4, 'B': 3, 'C': 4, 'D': 3, 'E': 5},\n",
       " 4: {'A': 3, 'B': 3, 'C': 1, 'D': 5, 'E': 4},\n",
       " 5: {'A': 1, 'B': 5, 'C': 5, 'D': 2, 'E': 1}}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"计算物品的相似矩阵\"\"\"\n",
    "similarity_matrix = pd.DataFrame(np.ones((len(items), len(items))), index=['A', 'B', 'C', 'D', 'E'], columns=['A', 'B', 'C', 'D', 'E'])\n",
    "\n",
    "# 遍历每条物品-用户评分数据\n",
    "\n",
    "for itemId in items:\n",
    "    for otheritemId in items:\n",
    "        vec_item = []         # 定义列表， 保存当前两个物品的向量值\n",
    "        vec_otheritem = []\n",
    "        #userRagingPairCount = 0     # 两件物品均评过分的用户数\n",
    "        if np.ndarray(itemId).all != np.ndarray(otheritemId).all:    # 物品不同 \n",
    "            \n",
    "            for userId in users:    # 遍历用户-物品评分数据\n",
    "                userRatings = users[userId]    # 每条数据为该用户对所有物品的评分， 这也是个字典\n",
    "                \n",
    "                if itemId in userRatings and otheritemId in userRatings:   # 用户对这两个物品都评过分\n",
    "                    #userRagingPairCount += 1\n",
    "                    vec_item.append(userRatings[itemId])\n",
    "                    vec_otheritem.append(userRatings[otheritemId])\n",
    "            \n",
    "            # 这里可以获得相似性矩阵(共现矩阵)\n",
    "            similarity_matrix[itemId][otheritemId] = np.corrcoef(np.array(vec_item), np.array(vec_otheritem))[0][1]\n",
    "            #similarity_matrix[itemId][otheritemId] = cosine_similarity(np.array(vec_item), np.array(vec_otheritem))[0][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"得到与物品5相似的前n个物品\"\"\"\n",
    "n = 2\n",
    "similarity_items = similarity_matrix['E'].sort_values(ascending=False)[:n].index.tolist()       # ['A', 'D']\n",
    "\n",
    "\"\"\"计算最终得分\"\"\"\n",
    "base_score = np.mean(np.array([value for value in items['E'].values()]))\n",
    "weighted_scores = 0.\n",
    "corr_values_sum = 0.\n",
    "for item in similarity_items:  # ['A', 'D']\n",
    "    corr_value = similarity_matrix['E'][item]            # 两个物品之间的相似性\n",
    "    mean_item_score = np.mean(np.array([value for value in items[item].values()]))    # 每个物品的打分平均值\n",
    "    weighted_scores += corr_value * (users[1][item]-mean_item_score)      # 加权分数\n",
    "    corr_values_sum += corr_value\n",
    "final_scores = base_score + weighted_scores / corr_values_sum\n",
    "print('用户Alice对物品5的打分: ', final_scores)\n",
    "user_df.loc[1]['E'] = final_scores\n",
    "user_df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 算法评估\n",
    "\n",
    "### 1.召回率\n",
    "\n",
    "R是推荐的东西，T是用户真正喜欢的东西\n",
    "\n",
    "$$\\operatorname{Recall}=\\frac{\\sum_{u}|R(u) \\cap T(u)|}{\\sum_{u}|T(u)|}$$\n",
    "\n",
    "用户真实购买或者看过的影片里面， 我模型真正预测出了多少， 这个考察的是模型推荐的一个全面性。\n",
    "\n",
    "### 2.准确率\n",
    "\n",
    "$$\\operatorname{Precision}=\\frac{\\sum_{u} \\mid R(u) \\cap T(u)|}{\\sum_{u}|R(u)|}$$\n",
    "\n",
    "在我推荐的所有物品中， 用户真正看的有多少， 这个考察的是我模型推荐的一个准确性。\n",
    "\n",
    "\n",
    "实际应用中，应该综合考虑全面性和准确率的平衡。准确率高了，总数就可能减少，导致全面性减小。\n",
    "\n",
    "### 3.覆盖率\n",
    "\n",
    "覆盖率反映了推荐算法发掘长尾的能力， 覆盖率越高， 说明推荐算法越能将长尾中的物品推荐给用户。\n",
    "\n",
    "$$\\text { Coverage }=\\frac{\\left|\\bigcup_{u \\in U} R(u)\\right|}{|I|}$$\n",
    "\n",
    "### 4.新颖度\n",
    "\n",
    "用推荐列表中物品的平均流行度度量推荐结果的新颖度。 如果推荐出的物品都很热门， 说明推荐的新颖度较低。 由于物品的流行度分布呈长尾分布， 所以为了流行度的平均值更加稳定， 在计算平均流行度时对每个物品的流行度取对数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 权重改进\n",
    "\n",
    "权重改进集中在热门商品和活跃用户上，通过调整对热门物品的惩罚吗，来防止所有物品都与热门商品相关度过高；活跃用户对物品相似度的贡献应该小于不活跃用户。\n",
    "\n",
    "## 算法的不足之处\n",
    "\n",
    "协同过滤算法存在的问题之一就是泛化能力弱， 即协同过滤无法将两个物品相似的信息推广到其他物品的相似性上。 导致的问题是热门物品具有很强的头部效应， 容易跟大量物品产生相似， 而尾部物品由于特征向量稀疏， 导致很少被推荐\n",
    "\n",
    "                A[0 0 0 1 1 0 1 0 1 0 1 ]  \n",
    "                B[0 1 0 0 0 0 0 0 0 0 0 ]  \n",
    "                C[0 0 1 0 0 0 0 0 0 0 0 ]  \n",
    "                D[1 1 1 1 1 1 1 1 1 0 1 ]  \n",
    "\n",
    "| |A|B|C|D|\n",
    "|--|--|--|--|--|\n",
    "|A|-|0.00|0.00|0.71|\n",
    "|B|0.00|-|0.00|0.18|\n",
    "|C|0.00|0.00|-|0.18|\n",
    "|D|0.71|0.18|0.18|-|\n",
    "\n",
    "A, B, C, D是物品， 看右边的物品共现矩阵， 可以发现物品D与A、B、C的相似度比较大， 所以很有可能将D推荐给用过A、B、C的用户。 但是物品D与其他物品相似的原因是因为D是一件热门商品， 系统无法找出A、B、C之间相似性的原因是其特征太稀疏， 缺乏相似性计算的直接数据。 所以这就是协同过滤的天然缺陷：推荐系统头部效应明显， 处理稀疏向量的能力弱。\n",
    "\n",
    "**其他不足之处**\n",
    "\n",
    "协同过滤的特点就是完全没有利用到物品本身或者是用户自身的属性， 仅仅利用了用户与物品的交互信息就可以实现推荐，比较简单高效， 但这也是它的一个短板所在， 由于无法有效的引入用户年龄， 性别，商品描述，商品分类，当前时间，地点等一系列用户特征、物品特征和上下文特征， 这就造成了有效信息的遗漏，不能充分利用其它特征数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
